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Abstract 

In  recent  years,  information  technology  has  advanced  at  a  pace  that  few  would 
have  anticipated.  It  has  been  estimated  that  the  computing  power  of  the  modem  desktop 
computer  has  been  increasing  at  the  rate  of  1000%  per  decade.  In  combination  with  the 
development  of  personal  computers,  the  advent  of  networks  and  the  world  wide  web 
provide  unprecedented  access  to  information  and  computing  power.  However,  the 
problem  of  developing  useful  user  interfaces  remains  a  problem.  In  many  military  and 
commercial  settings,  the  increased  computing  power  offered  by  cmrent  information 
technology  remains  unexploited  because  of  user  interfaces  that  are  difficult  to  use. 

This  thesis  reports  on  the  examination  of  constructs  related  to  user  acceptance  of 
information  systems  from  two  disciplines,  human-computer  interaction  (HCI)  and 
management  information  systems  (MIS).  More  specifically,  research  was  conducted  to 
evaluate  the  possibility  of  overlap  between  the  two  divergent  fields.  The  Technology 
Acceptance  Model  (TAM)  from  MIS  research  was  examined  in  light  of  the  HCI 
constructs  of  efficiency,  effectiveness,  and  satisfaction.  Of  interest  was  the  impact  of 
efficiency,  effectiveness,  and  satisfaction  on  the  formulation  of  user  perceptions  of  ease 
of  use. 

The  empirical  data  suggests  that  satisfaction  plays  a  major  role  in  the  formulation 
of  user  perceptions  (p<0.01),  while  the  role  of  efficiency  and  effectiveness  are  minimal. 
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TECHNOLOGY  ACCEPTANCE:  A  FUSION  OF  HUMAN-COMPUTER 
INTERACTION  AND  MANAGEMENT  INFORMATION  SYSTEMS  CONSTRUCTS 

I.  Introduction 

Background 

Acceptance  of  technology  by  users  has  been  studied  for  more  than  a  decade.  The 
attempt  to  capture  and  define  the  characteristics  of  technology  that  promote  acceptance, 
or  define  a  model  that  can  be  used  to  successfully  predict  acceptance  of  technology  has 
been  at  the  heart  of  this  research  (Fishbein  and  Ajzen,  1975;  Davis,  1989;  Venkatesh  and 
Davis,  1996).  In  general,  user  acceptance  has  been  “defined  as  the  demonstrable 
willingness  within  a  user  group  to  employ  information  technology  for  tasks  it  is  designed 
to  support”  (Dillon  &  Morris,  1996). 

Interest  in  user  acceptance  is  widespread,  finding  appeal  at  many  levels  of  the 
information  technology  (IT)  industry  and  throughout  the  business  community.  Business 
enterprises  have  a  natural  interest  because  they  must  make  sizable  investments  in  IT  as  a 
means  of  obtaining  competitive  advantage  through  reduction  of  costs,  or  reduction  of 
processing  or  production  time.  The  current  trend  in  business  is  for  greater  and  greater 
reliance  on  IT  to  improve  effectiveness  and  efficiency,  and  ultimately,  the  bottom  line. 
From  the  perspective  of  business  enterprises  that  utilize  IT,  such  investments  should 
procure  new  capabilities  or  improvements  in  existing  capabilities  considered  necessary 
for  business  competition.  However,  the  consequential  benefits  of  such  investments  are 
not  guaranteed.  For  various  reasons,  IT  does  not  always  provide  the  desired  advantages 
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or  improvements;  in  fact,  an  ill-conceived  IT  program  can  have  negative  effects  on  an 
organization  (Landauer,  1995). 

There  are  many  causes  for  ineffectual  IT  investment  and  employment,  such  as 
imrealistic  expectations  of  IT,  poorly  matched  hardware  and  software,  insufficient 
investment  of  time  and/or  money,  and  inadequate  training.  The  lack  of  user  acceptance 
that  may  accompany  any  of  these  unsatisfactory  conditions  can  be  the  death  knell  of  a 
system.  No  matter  how  capable  a  system  is,  if  users  reject  the  system,  the  capability  is 
not  exploited  and  the  benefit  of  the  investment  is  not  experienced  (Davis,  Bagozzi,  and 
Warshaw,  1989).  High  costs  are  associated  with  the  innovative  use  of  IT,  and  investment 
in  systems  that  are  not  accepted  by  the  intended  users  must  be  avoided.  Innovative  IT 
applications  can  be  expensive,  but  even  the  use  of  standard  systems  employing 
conventional  configurations  and  software  can  impose  a  significant  financial  burden  on  a 
small  or  medium  sized  enterprise.  Indeed,  avoidance  of  the  negative  consequences  of 
inadequate  IT  implementations  motivates  much  of  the  interest  in  technology  acceptance. 

In  addition  to  the  need  for  user  acceptance  in  the  business  arena,  investigation  into 
user  acceptance  is  also  motivated  by  competition  within  the  IT  industry.  Within  the 
technology  industry  competition  remains  fierce,  further  motivating  the  necessity  to 
develop  systems  that  are  readily  received.  User  acceptance  of  software  products  can 
mean  success  or  failure  for  software  firms.  Often  the  distinguishing  characteristic 
between  competing  products,  especially  in  consumer  electronics  and  software,  is  the  ease 
of  use  of  the  products.  As  a  result  ease  of  use  is  of  critical  and  increasing  importance. 

According  to  March  (1994:1 1),  user-centered  design  is  expanding  traditional 
design  by  encompassing  cognitive  and  emotional  aspects  of  user  response: 
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Many  companies  will  also  have  to  rethink  and  reorganize  the  design  process. .  .to 
ensure  that  usability  is  designed  into  the  product  from  the  outset.  With  the  gap 
between  competing  products  narrowing  in  terms  of  performance  and  quality,  the 
experience  that  a  product  delivers  is  rapidly  becoming  the  key  to  offering 
distinctive  value  to  the  customer. 

In  the  national  defense  arena  usability  is  also  taking  on  increased  importance.  The 
vmique  needs  of  the  military  have  led  to  the  development  of  an  acquisition  industry  that 
produces  specialized  equipment  for  military  applications.  In  this  setting,  it  is  critical  that 
users  accept  and  use  provided  equipment  for  its  intended  purpose. 

Management  Information  Systems  (MIS)  Perspective 

From  a  theoretical  perspective,  the  Technology  Acceptance  Model  (Davis, 
Bagozzi,  and  Warshaw  1989)  represents  an  attempt  to  integrate  external  variables,  user 
attitudes  and  perceptions,  and  system  features  into  a  model  that  can  be  used  by 
practitioners  and  researchers  to  explain  and  predict  actual  system  usage  (see  Figure  1). 
The  Technology  Acceptance  Model  (TAM)  has  subsequently  been  supported  by  a  large 
body  of  research  results  and  has  become  the  most  widely  accepted  model  of  acceptance  in 
the  IT  arena  (e.g.,  see  Adams,  Nelson,  and  Todd,  1992,  Dillon  and  Morris,  1996;  and 
Venkatesh  and  Davis,  1996).  Briefly,  TAM  posits  that  external  factors  impact  the  user’s 
internal  beliefs  and  attitudes  towards  technology,  and  beliefs  and  attitudes  influence  the 
user’s  intention  to  use  or  reject  the  subject  system. 
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Figure  1.  The  Technology  Acceptance  Model 


Perceived  Ease  of  Use  (EOU)  is  a  construct  in  the  TAM  that  refers  to  the  user’s 
belief  that  use  of  the  target  system  will  be  free  of  effort  (Davis,  Bagozzi,  and  Warshaw 
1989:985).  Behavioral  Intent  (BI)  refers  to  the  strength  of  the  intention  to  perform  a 
specified  action,  in  this  case,  use  the  system.  The  current  state  of  imderstanding  on  the 
relationship  between  EOU  and  BI  indicates  that  a  strong  positive  correlation  exists 
between  EOU  and  BI.  That  is,  the  easier  a  system  is  to  use,  the  more  likely  actual  use 
will  occur.  However,  less  is  known  about  the  antecedents  of  EOU.  What  is  known  is  that 
EOU  is  affected  by  a  variety  of  factors,  both  external  and  internal.  External  factors  are 
associated  with  the  environment,  and  internal  factors  are  within  the  user.  More 
specifically,  the  external  factors  include  such  items  as  computer  interface,  training,  and 
documentation.  Internal  factors  include  computer  self-efficacy,  enjoyment,  and  anxiety. 

Figure  1  depicts  the  relationship  between  external  variables  and  EOU.  A  partial 
list  of  external  variables  was  provided  in  the  preceding  paragraph.  All  of  these  factors 
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contribute  differently  to  forming  EOU.  It  is  also  likely  that  other  non-specified  external 
factors  affect  the  formulation  of  EOU,  but  the  current  state  of  knowledge  does  not  define 
all  possible  factors. 

Human  Computer  Interaction  (HCI)  Perspective 

It  is  important  to  note  that  contributions  in  defining  the  factors  that  contribute  to 
EOU  have  been  made  by  both  Management  Information  Systems  (MIS)  and  Human 
Computer  Interaction  (HCI)  researchers.  In  particular,  HCI  efforts  have  done  much  to 
identify  and  develop  quantifiable  measures  of  external  variables  (Shneiderman,  1992:2- 
19).  Although  the  HCI  research  has  historically  focused  on  the  definition  and 
measurement  of  usability,  while  subject  MIS  research  has  focused  on  riser  perceptions,  it 
may  be  possible  to  define  theoretical  ground  common  to  both  the  MIS  and  HCI  views  of 
user  acceptance.  It  is  also  possible  that  the  separate  pursuits  for  understanding  in  the 
area  of  user  acceptance  have  led  MIS  and  HCI  researchers  to  common  ground,  and  that 
the  divergent  terminology  and  perspective  of  the  two  disciplines  have  obscured  the 
conunonality.  In  TAM,  acceptance  is  operationally  defined  by  actual  system  usage.  In 
turn,  acceptance  is  determined  by  a  wide  range  of  attitudes,  perceptions,  and  external 
variables.  It  is  reasonable  to  conclude  that  external  variables  such  as  effectiveness  and 
efficiency  play  an  important  role  in  forming  perceptions  of  EOU.  Effectiveness  and 
efficiency  are  actually  determinants  borrowed  from  the  HCI  definition  of  usability. 
Within  the  HCI  perspective,  effectiveness  and  efficiency  are  usability  variables  that  can 
be  operationally  defined  and  measured  to  produce  a  quantifiable  definition  of  usability 
(Dillon  and  Morris,  1996:20).  It  may  be  postulated  that  subjective  usability  mediates 
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between  objective  usability  and  use  in  the  same  manner  that  EOU  mediates  between 
external  variables  and  actual  use. 

It  is  possible  that  EOU  perceptions  are  formed  through  an  experience-based 
cognitive  evaluation  of  the  target  system.  This  does  not  imply  that  a  user  devises  a 
formal  measure  of  efficiency  and  effectiveness,  rather  that  the  user  formulates  critical 
qualitative  measures  of  efficiency  and  effectiveness  as  by-products  of  objective 
experience  with  the  target  system.  It  is  also  possible  that  satisfaction,  an  additional 
component  of  the  HCI  usability  construct,  contributes  to  perceived  EOU.  It  may  be 
postulated  that  the  EOU  perceptions  are  indeed  formed  by  a  cognitive  evaluation  of  the 
target  system.  Specifically,  it  may  be  possible  that  EOU  perceptions  are  shaped  through 
the  external  determinants  that  are  summed  in  efficiency  and  effectiveness  measures. 

Research  Question 

At  a  theoretical  level  this  research  deals  with  the  question  of  predicting  user 
acceptance.  The  usability  constructs  of  efficiency,  effectiveness,  and  satisfaction  largely 
measure  the  ability  of  system  users  to  complete  tasks.  System  design  features  affect 
system  usefulness  and  also  lead  to  EOU  perceptions  (Davis,  Bagozzi,  and  Warshaw, 
1989).  But  there  remains  a  more  fundamental  issue:  Will  users  voluntarily  use  a  system 
to  accomplish  tasks?  In  particular,  this  study  focuses  on  the  role  of  objective  usability  in 
forming  EOU  perceptions. 

The  problem  dealt  with  in  this  research  can  be  summarized  as  follows:  Can  it  be 
shown  that  the  MIS  concepts  of  external  variable,  EOU,  and  actual  use  correspond  with 
the  HCI  concepts  of  objective  usability,  subjective  usability,  and  actual  use? 
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This  thesis  reports  on  an  empirical  study  conducted  to  verify  a  correspondence 
between  the  concepts  of  external  variables  and  objective  usability;  and  EOU  and 
subjective  usability.  During  and  subsequent  to  a  word  processing  task,  two  groups  were 
measured  according  to  constructs  that  rise  from  the  concepts  of  usability,  satisfaction,  and 
perceptions  about  use.  A  review  of  the  relevant  literature  will  be  followed  by  a  detailed 
presentation  of  the  methodology  used  in  this  research.  This  will  be  followed  by  a 
presentation  of  the  statistical  analysis  and  a  discussion  of  the  results. 
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II.  Literature  Review 


Introduction 

The  intention  of  this  review  is  to  collect  and  examine  the  relevant  knowledge  and 
research  stream  related  to  formulation  of  user  perceptions  and  usability  issues. 

Computer  processing  power  continues  to  increase  at  a  rapid  rate.  In  fact,  computing 
power  has  historically  increased  at  about  1000%  per  decade  (Davis,  Bagozzi,  and 
Warshaw,  1989:982).  This  continuous  improvement  in  computing  makes  possible  the 
expanding  role  of  the  computer  at  work  and  at  home.  Modem  manufacturing  and 
services  have  been  redesigned  to  capitalize  on  the  control  and  data  analysis  capabilities  of 
the  computer.  As  an  example  of  the  move  to  greater  commercial  reliance  on  computers, 
McDonald’s  Restaurants  is  currently  transitioning  to  ARCH  (Automated  Restamant  Crew 
Helper),  a  computer-based  system  that  will  automate  food  preparation,  inventory  control, 
transaction  processes,  and  even  predict  future  demands.  The  system  will  sharply  reduce 
manpower  requirements  and  improve  efficiency  of  service  operations  (Murray,  1993). 
The  traditional  use  of  computerized  systems  in  manufacturing  is  now  being 
complemented  by  the  development  of  computerized  systems  for  use  in  service  industries. 
Expansion  of  computer  use  in  the  service  industries  portends  an  even  greater  role  for 
computers  in  the  future. 

The  trend  toward  greater  and  more  widespread  computer  utilization  may  create 
the  impression  that  user  acceptance  is  a  mature  science,  when,  in  fact,  information  system 
(IS)  researchers  and  designers  are  still  seeking  knowledge  and  methods  that  will  lead  to 
the  ability  to  predict  user  acceptance,  e.g.,  Davis,  Bagozzi,  and  Warshaw,  1989; 
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Venkatesh  and  Davis,  1996.  User  acceptance  has  been  defined  as  “the  demonstrable 
willingness  within  a  user  group  to  employ  IT  for  the  tasks  it  is  designed  to  support” 
(Dillon  and  Morris,  1996:4).  Understanding  the  perceptions  and  motivations  that  precede 
acceptance  or  rejection  of  a  system  has  long  been  problematic  for  IS  professionals 
(Swanson,  1988). 

Researchers  have  examined  many  aspects  of  the  user-system  relationship  to  gain 
insight  into  user  acceptance.  Ives  and  Olson  (1984),  Haitwick  and  Barki  (1994),  and 
McKeen,  Guimaraes,  and  Wetherbe  (1994)  investigated  the  correlation  between  user 
involvement  in  the  development  process  and  user  acceptance.  In  addition,  others  have 
examined  how  the  cognitive  style  of  the  individual  user  has  been  considered  as  a  factor  in 
acceptance  (Huber,  1983).  Additionally,  the  effect  of  user  beliefs  and  attitudes  on  actual 
system  use  has  received  much  attention  (Ives,  Olson,  and  Baroudi,  1983:785-793;  Davis 
and  others,  1989).  Furthermore,  irmovation  diffusion  theory  has  also  been  referenced  to 
illuminate  the  acceptance  of  information  technology  and  determinants  of  technology 
adoption  within  user  communities  (Rogers,  1995). 

MIS  Conceptualization  of  Acceptance 

As  yet,  no  single  factor  has  emerged  as  a  reliable  predictor  of  user  acceptance 
(Dillon  and  Morris,  1996:5).  But  this  is  not  to  say  that  understanding  of  user  acceptance 
has  not  been  furthered.  Rather  than  a  single-factor  predictor  of  user  acceptance,  multi¬ 
factor  theories  have  risen  from  user  acceptance  research. 
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Theory  of  Reasoned  Action  (TRA).  The  Theory  of  Reasoned  Action  (Ajzen  and 
Fishbein,  1980)  is  a  widely  studied  model  that  focuses  on  the  determinants  of  intended 
behaviors.  The  Theory  of  Reasoned  Action  (TRA)  posits  that  an  individual’s  behavioral 
intent  (BI)  or  intention  to  act  is  a  good  predictor  of  actual  behaviors.  BI  is  influenced  by 
the  individual’s  attitude  (A)  and  subjective  norm  (SN).  Figure  2  shows  a  diagram  of 
TRA. 


Figure  2.  The  Theory  of  Reasoned  Action  (TRA)  Model 


Beliefs  and  evaluations  (upper  left  box)  are  defined  as  the  level  of  an  individual’s 
subjective  confidence  that  a  given  action  will  produce  a  given  result,  and  an  evaluation  of 
the  desirability  of  that  result  (Davis  and  others,  1 989:984).  For  example,  individuals 
needing  to  solve  simultaneous  equations  may  believe  that  using  a  programmable 
calculator  vvdll  be  faster  and  more  flexible  than  using  a  spreadsheet  program  loaded  on 
their  desktop  computers.  The  increased  flexibility  will  allow  them  to  complete  work 


10 


while  riding  the  subway,  instead  of  spending  after-hours  time  in  the  office,  hence  arriving 
home  earlier.  The  belief  that  the  action  will  lead  to  the  result  and  the  evaluation  of  the 
result  (arriving  home  earlier)  will  affect  the  Attitude  Toward  Behavior.  Attitude  Toward 
Behavior  denotes  the  consequential  subjective  attitude  formed  by  belief  in  the  likelihood 
and  desirability  of  the  consequences  of  a  given  action.  Attitude  Toward  Behavior  is  one 
of  the  two  determinants  of  Behavioral  Intent  (BI)  (Ajzen  and  Fishbein,  1980). 

According  to  Davis  and  others  (1989:984), 

A  partic\ilarly  helpful  aspect  of  TRA  from  an  IS  perspective  is  its  assertion  that 
any  other  factors  that  influence  behavior  do  so  only  indirectly  by  influencing  A, 
SN,  or  their  relative  weights.  TRA  captures  the  internal  psychological  variables 
through  which  numerous  external  variables  studied  in  IS  research  achieve  their 
influence  on  user  acceptance. 

A  significant  body  of  research  indicates  that  A  and  SN  mediate  the  effect  of 
external  variables  on  BI  (Bagozzi,  1984;  Warshaw  and  Davis,  1985,  1986). 

Technology  Acceptance  Model  (TAM).  Although  TRA  has  received  reliable 
empirical  support,  it  is  a  general  model.  Davis  (1989)  adapted  TRA  for  use  in  IS 
research,  especially  for  study  in  the  area  of  user  acceptance.  The  adaptation,  referred  to 
as  the  Technology  Acceptance  Model  (TAM)  is  pictured  below. 
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Figure  3.  Technology  Acceptance  Model  (TAM) 


Research  has  confirmed  that  Behavioral  Intent  (BI)  is  the  single  best  predictor  of 
actual  use  (Taylor  and  Todd,  1995:144-176).  TAM  describes  the  relationships  between 
the  attitudes  and  beliefs  that  form  BI.  In  TAM,  BI  is  modeled  as  a  fimction  of  Perceived 
Usefulness  (U)  and  Perceived  Ease  of  Use  (EOU);  and  TAM  posits  that  U  and  EOU  are 
primary  factors  in  computer  acceptance  behaviors.  U  is  defined  as  the  degree  to  which 
the  prospective  user  believes  that  use  of  a  specific  application  system  will  increase  his  or 
her  performance.  EOU  is  defined  as  the  degree  to  which  the  prospective  user  believes 
that  use  of  a  specific  system  will  be  free  of  effort  (Davis  and  others,  1989:985;  Dillon  and 
Morris,  1996:10).  Furthermore,  it  is  posited  in  TAM  that  external  variables  (EV),  which 
may  include  such  items  as  system  features  (mice,  menus,  graphical  user  interface), 
training,  documentation  and  user  support,  can  also  influence  EOU  (Davis  and  others, 
1989:987-988). 
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TAM  has  emerged  as  an  effective  tool  in  predicting  user  acceptance.  Extensive 
empirical  support  for  the  model  exists  e.g.,  Adams,  Nelson,  and  Todd,  1992;  Chin  and 
Todd,  1995;  Davis  and  Venkatesh,  1995;  and  Davis  and  Venkatesh,  1996.  While  a 
significant  body  of  research  supports  the  validity  and  robustness  of  TAM,  its  generality 
has  not  yet  provided  adequate  predictive  knowledge  for  use  by  system  designers.  In 
recent  work,  Davis  recognizes  that  this  outcome  “while  being  very  powerful  in  helping  us 
predict  acceptance,  one  of  the  limitations  of  TAM  is  that  it  does  not  help  understanding 
and  explain  acceptance  in  ways  that  guide  development”  (Venkatesh  and  Davis, 
1996:472).  Further,  according  to  Venkatesh  and  Davis  (1996:452),  the  preponderance  of 
TAM  research  has  focused  on  U  and  EOU,  with  little  research  dedicated  to  understanding 
the  determinants  of  EOU. 

Antecedents  of  EOU.  Figure  3  shows  EOU  modeled  as  a  function  of  External 
Variables  (EV).  According  to  Davis  (1989:987-989),  a  key  purpose  of  TAM  is  to  provide 
a  basis  for  tracing  the  impact  of  external  factors  on  internal  attitudes.  Little  research  has 
focused  specifically  on  the  antecedents  and  determinants  of  EOU,  namely,  the  EVs. 
However,  the  antecedents  that  have  been  suggested  include  Perceived  Behavioral  Control 
(Ajzen,  1985;  Mathieson,  1991),  Enjoyment,  Computer  Self-Efficacy,  Objective 
Usability,  Computer  Anxiety  (Venkatesh  and  Davis,  1996;  Venkatesh,  1998  under 
review).  A  pictorial  representation  of  these  antecedents  is  presented  in  Figure  4. 
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Figure  4.  Antecedents  of  EOU 


Perceived  Behavioral  Control  (PBC)  is  a  construct  that  describes  the  relationship 
between  enabling  aspects  of  the  environment  (knowledge,  available  resources,  and 
opportumties)  and  behavior  (Azjen,  1985:1 1-39).  PBC  is  determined  by  control  beliefs 
and  perceived  facilitation.  Control  beliefs  pertain  to  the  perceptions  of  availability  of 
knowledge,  resources,  and  opportumties.  Perceived  facilitation  relates  to  the  perceived 
importance  of  these  on  the  desired  outcomes. 

With  respect  to  information  systems,  enjoyment  has  been  defined  as  “the  extent  to 
which  the  activity  of  using  the  computer  is  perceived  to  be  enjoyable  in  its  own  right. 
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apart  from  any  performance  consequences  that  may  be  anticipated”  (Davis  and  others, 
1992:1 1 13).  Further,  recent  research  by  Venkatesh  (1998)  suggests  that  higher  levels  of 
enjoyment  lead  to  lowered  perceptions  of  effort — or  higher  levels  of  perceived  ease  of 
use. 

Self-EfBcacy,  defined  as  “judgments  of  how  well  one  can  execute  courses  of 
action  required  to  deal  with  prospective  situations”  (Bandura,  1982)  is  important  to  the 
formulation  of  EOU.  In  general  applications,  it  has  been  demonstrated  that  self-efficacy 
influences  the  decisions  to  use  a  particular  computer,  independent  of  the  perceived 
instrumental  value  of  the  system  (Hill,  Smith  and  Maim,  1987).  Specific  to  TAM, 
research  has  supported  the  proposition  that  Computer  Self-Efficacy,  the  application  of 
self-efficacy  theory  to  computer  systems,  is  an  antecedent  of  EOU  (V enkatesh  and  Davis, 
1996). 

Objective  Usability  has  been  proposed  as  an  antecedent  of  EOU  (Venkatesh  and 
Davis,  1996).  In  the  MIS  vernacular.  Objective  Usability  represents  an  objective  measure 
of  system  characteristics.  Accomplished  through  operationalizing  the  facet  of  interest. 
Objective  Usability  provides  a  means  for  comparisons  among  systems.  The  construct  of 
Objective  Usability  has  long  existed  in  the  Human-Computer  Interaction  context 
(Shackel  and  Richardson,  1991:24-25),  where  usability  inspection  (Mark  and  Nielsen, 
1994)  has  been  evolved  into  a  recognized  methodology.  The  high-level  objectives  of 
understanding  usability  issues  are  to  identify  potential  usability  problems  and  to  develop 
design-relevant  knowledge  that  is  useful  in  improving  usability  (Mark  and  Nielsen, 
1994:1.4) 
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Computer  Anxiety  consists  of  the  fear  or  apprehension  that  individuals  may 
experience  when  confronted  with  possibility  of  using  a  computer  (Simonson,  Mauer, 
Montag-Torardi,  and  Whitaker,  1987:232).  It  has  been  postulated  that  computer  anxiety 
negatively  affects  perceived  EOU  (Venkatesh,  1998  under  review). 

Human-Computer  Interaction  Perspective 

The  field  of  Human-Computer  Interaction  (HCI),  a  subset  of  the  Human-System 
Interaction  discipline,  focuses  on  computer  use  issues,  to  include  Human  Factors  and 
Ergonomics  (Shackel  and  Richardson,  1991 :12-13).  System  acceptance  by  users  is  one 
of  the  central  themes  and  research  focuses  of  HCI.  By  analytical  consideration  of  all 
aspects  of  a  system  with  which  users  interact,  HCI  seeks  to  understand  and  enumerate  the 
factors  that  contribute  to  user  acceptance  of  systems  (Nielsen,  1993:24-25).  The 
theoretical  construct  of  system  acceptance  has  been  encapsulated  by  Shackel  (1991 :22)  in 
an  acceptability  equation.  The  equation  states: 

{UTILITY  (will  it  do  what  is  needed  fimctionally?)  -i-  USABILITY  (will  the  users 
actually  work  it  successfully?)  +  LIKEABILITY  (will  the  users  feel  it  is 
suitable?)} 

must  be  balanced  in  a  trade-off  against 
COST  (what  are  the  capital  and  running  costs?,  what  are  the  social  and 
organizational  consequences?) 

to  arrive  at  a  decision  about 

ACCEPTABILITY  (on  balance  the  best  possible  alternative  for  purchase) 


16 


It  should  be  noted  that  this  paradigm  places  the  same  importance  on  usability  as  it 
places  on  functionality  requirements.  It  must  be  understood  that  simply  providing  the 
user  with  the  facilities  to  perform  tasks  helps  little  if  the  user  does  not  employ  the  system 
effectively,  cannot  find  the  functions,  or  avoids  systems  use  for  various  reasons. 

Usability  Defined.  A  variety  of  theoretical  and  working  definitions  for  usability  can 
be  found  in  the  HCI  literature.  Representative  definitions  follow. 

-Usability  typically  refers  to  the  following  three  characteristics:  speed,  accuracy,  and 
satisfaction  (Hoffer,  George,  and  Valacich,  1996:490). 

-User  friendliness  should  be  replaced  with  “Time  to  learn.  Speed  of  performance.  Rate 
of  errors  by  users.  Retention  over  time,  and  Subjective  satisfaction”  (Shneiderman, 
1992:15-19). 

—“The  capability  in  human  functional  terms  to  be  used  easily  and  effectively  by  the 
specified  range  of  users,  given  specified  training  and  user  support,  to  fulfill  the  specified 
range  of  tasks ,  within  the  specified  range  of  environmental  scenarios”  (Shackel, 

1991:24). 

-And  according  to  Nielsen  (1993:26),  “Usability  has  multiple  components  an  is 
traditionally  associated  with  these  five  usability  attributes:  Leamability,  Efficiency, 
Memorability,  Errors,  and  Satisfaction.” 

—“According  to  ISO  9241  pt.  1 1  the  usability  of  an  application  refers  to  the  effectiveness, 
efficiency,  and  satisfaction  with  which  specific  users  who  are  peirforming  specific  tasks  in 
specific  environments  can  use  an  application”  (Dillon  and  Morris,  1996:20). 


At  the  heart  of  the  system  acceptance  model,  usability  issues  focus  on  the  central 
user-centered  design  question,  “Will  they  use  it?”  Figure  5  is  an  adaptation  of  an 
acceptability  model  that  illustrates  the  role  of  usability  in  system  acceptance  (Nielsen, 
1993:25).  The  model  presents  the  attributes  of  system  acceptability  and  the  relationships 
between  the  varied  factors. 


Figure  5.  A  Model  of  the  Attributes  of  System  Acceptability 

The  emergence  of  usability  as  a  critical  design  criteria  is  emphasized  by  the 
International  Standards  Orgamzation  (ISO)  introduction  of  a  new  international  standard 
(ISO  9241)  that  covers  usability  in  software  and  systems  (Dillon,  1994).  A  large  volume 
of  work  exists  (Nielsen,  1993;  Shneiderman,  1992;  Shackel  and  Richardson,  1991) 
explicating  the  relationship  between  usability  and  user  interface,  indicating  that  the  user’s 
objective  experiences  with  a  system  lead  to  the  formation  of  perceptions  about  the 
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system.  At  this  point  it  is  useful  to  recall  that  TAM  describes  the  mediating  effect  EOU 
has  on  the  relationship  between  external  variables  and  perceptions  of  the  system. 
According  to  Davis,  et  al,  (1989),  system  design  aspects,  such  as  the  user  interface,  lead 
to  EOU  perceptions. 

Integrating  MIS  and  HCI  Research 

As  stated  in  chapter  one,  it  is  the  purpose  of  this  thesis  to  relate  the  HCI  system 
acceptability  model  and  the  MIS  technology  acceptance  model.  Specifically,  this  thesis 
proposes  to  correlate  the  theoretical  constructs  of  usability  firom  the  HCI  model  and  EOU 
firom  the  MIS  model.  Further,  it  will  investigate  how  utility  from  the  HCI  model 
correlates  with  U  (usefulness)  firom  the  MIS  model.  Thus,  given  these  alternatives 
perspectives,  the  following  proposition  is  offered: 

The  user’s  cognitive  experience  with  the  technology,  in  terms  of  effectiveness, 
efficiency,  and  satisfaction  significantly  influence  the  formation  of  EOU  perceptions. 
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III.  Method 


Research  Approach 

This  chapter  presents  details  of  the  experiment  and  data  gathering.  The 
experiment  was  conducted  in  a  laboratory  setting  at  a  large  university.  Two  popular  word 
processing  software  packages  were  used  to  assess  objective  usability  and  perceived  ease 
of  use.  The  following  sections  describe  the  sample,  design  and  measures  for  this  study  in 
detail. 


Sample.  The  participants  in  the  experiment  were  freshmen  and  sophomores 
enrolled  in  a  required  class  at  a  large,  midwestem  imiversity.  Students  were  required  to 
participate  in  graduate  research  to  fulfill  course  requirements.  Although  participation  in 
research  was  a  requirement,  students  were  able  to  choose  from  a  variety  of  research 
projects. 

Design.  The  two  word  processing  software  packages  used  in  the  experiment  were 
Word  for  Windows  and  WordPerfect  5.1  (DOS  version).  These  two  software  packages 
were  chosen  because  of  their  popularity,  and  because  both  word  processors  were  already 
installed  in  the  university  computer  labs.  The  participants  were  prescreened  to  sort  the 
participants  into  two  groups;  Word  users  and  WordPerfect  users.  All  participants  haH 
previous  experience  with  one  or  both  word  processors.  Participants  with  Word 
experience  were  placed  in  the  group  that  used  Word  in  the  experiment,  and  participants 
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with  WordPerfect  experience  were  placed  in  the  group  that  used  WordPerfect  in  the 
experiment. 

The  participants  were  provided  with  a  hardcopy  of  a  two-page,  multi-paragraph 
text  document  that  incorporated  commonly  found  features  of  word  processing,  e.g., 
underlining,  bolded  text,  italics,  text  centering,  and  various  font  sizes.  A  text  file 
containing  only  the  text  (no  formatting,  spacing,  etc.)  of  the  document  was  provided  on  a 
floppy  disk.  Participants  were  instructed  to  use  the  word  processor  to  edit  the  text  file 
and  strive  to  produce  an  exact  copy,  or  copy  as  closely  as  possible  the  sample  document. 
Instructions  indicated  that  the  copy  should  incorporate  the  same  fonts,  features  and  text  of 
the  sample  document.  All  instructions  were  conveyed  verbally  and  each  participant 
received  written  instructions.  Additionally,  participants  were  given  the  following  hints  in 
the  written  instructions: 

-font  is  Univers 

-title  size  is  14  point,  everything  else  is  12  point 
-margin  is  1  inch  on  top,  bottom ,  left  and  right 
-all  double  and  triple  space  locations  were  identified 

Participants  were  allowed  to  work  at  their  own  paces.  Times  for  each  participant 
to  complete  the  task  was  recorded.  After  completion  of  the  document,  participants 
completed  questionnaires  designed  to  elicit  participants’  reactions  to  the  word  processor 
they  had  used.  The  questionnaire  is  discussed  in  the  next  section. 

Measures.  Consistent  with  the  literature  on  usability  assessment,  measures  of 
effectiveness,  efficiency,  and  satisfaction  were  gathered  to  assess  system  usability.  In 
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addition,  measures  of  EOU  and  U  from  the  MIS  literature  were  gathered  to  help 
understand  the  nomological  net  associated  with  the  usability  construct.  Reproduction  of 
the  text  page  was  graded  with  a  nine-point  criteria  to  measure  accuracy.  This 
measurement  leads  to  an  effectiveness  score.  The  questionnaire  contained  a  variety  of 
measuring  instruments  (see  Table  1).  Additionally,  the  reproduced  text  was  graded  as  an 
operationalized  measure  of  accuracy  and  task  completion. 


Table  1.  Measurement  Instruments 


Effectiveness 

Efficiency 

Satisfaction 

MIS 

Accuracy 

Time 

QUIS 

EOU 

Task  Completion 

TLX 

U 

The  reproduced  document  was  graded  using  a  nine-point  criteria  that  reflected  the 
accuracy  of  the  reproduction.  The  nine  criteria  used  were  not  weighted,  and  were  each 
worth  one  point.  The  highest  possible  score  was  9,  and  the  lowest  possible  score  weis  0. 
The  criteria  were: 

-selecting  the  correct  font  type 
-selecting  the  correct  font  size  for  the  title 
-selecting  the  correct  font  size  for  the  body 
-centering  the  title 
-bolding  the  headings 
-underlining  the  headings 
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-correct  use  of  italics 


-correct  line  spacing 
-correct  margins 

The  EOU  scale  (Davis,  1989)  consisted  of  four  items.  The  participants  indicated 
the  degree  of  EOU  on  a  7-point  Likert  scale.  A  response  of  1  indicated  strongly  disagree, 
4  indicated  neutral  (neither  agree  nor  disagree),  7  indicated  strongly  agree.  The  measure 
of  EOU  is  derived  by  adding  the  scores  on  each  of  the  EOU  items  and  dividing  by  four  to 
provide  an  overall  index  of  EOU. 

The  perceived  usefulness  scale  used  in  this  study  has  demonstrated  strong 
reliability  and  internal  consistency  (Davis,  1989).  The  U  scale  consists  of  four  items 
indicative  of  the  construct.  As  with  EOU,  the  participant  was  asked  to  indicate  the  degree 
of  U  on  a  7-point  Likert  scale.  A  response  of  1  indicated  strongly  disagree,  4  indicated 
neutral  (neither  agree  nor  disagree),  and  7  indicated  strongly  agree.  Individvial  measures 
of  U  were  derived  by  adding  the  scores  on  each  of  the  U  items  and  dividing  by  four  to 
provide  an  overall  index  of  U. 

Satisfaction  was  measured  by  coding  the  QUIS  into  the  questionnaire.  The  five 
components  of  satisfaction  measured  by  the  QUIS  were: 

-overall  reaction  to  the  system 

-screen 

-terminology 

-learning 

-system  capabilities 
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Individual  scores  for  the  five  components  were  obtained  by  averaging  the  component 
responses. 

Measurement  of  subjective  mental  workload  (SMW)  was  achieved  through  use  of 
the  TLX.  The  TLX  has  been  used  extensively  in  human  factors  research  and  is 
considered  to  be  one  of  the  best  measures  of  SMW  available  (Hancock  and  Meshkati, 
1988:139).  The  TLX  employs  a  pair-wise  comparison  of  six  dimensions.  The  six 
dimensions  are: 

-mental  demand 
-physical  demand 
-time  demand 
-performance 
-effort 

-frustration  level 

In  addition  to  the  pair-wise  dimensions,  TLX  employs  a  scaled  rating  of  each  of  the  five 
dimensions  with  ratings  ranging  from  0  to  100.  Calculation  of  the  mental  workload 
scores  involved  tallying  the  times  each  dimension  was  identified  in  the  pair-wise 
comparisons.  Each  dimension  could  be  identified  from  0  to  5  times.  The  pair-wise 
comparison  score  was  then  multiplied  by  the  corresponding  scaled  rating.  This  product 
led  to  a  sum  that  was  divided  by  1 5  to  produce  a  workload  score  (Hancock  and  Meshkati, 
1988:177). 


Statistical  Analysis.  The  data  collected  were  analyzed  using  a  variety  of  statistical 
techniques  including  descriptive  statistics,  normality  tests,  t-tests,  and  multiple 
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regression.  Descriptive  statistics  such  as  means  and  standard  deviations  are  used  to 
prepare  the  data  for  further  analysis  and  provide  a  general  idea  of  the  ranges  and  trends  of 
the  data.  Paired  t-tests  are  used  to  indicate  the  validity  of  the  samples  and  hence  the 
comparisons  of  the  samples.  Finally,  the  factors  described  in  Chapters  I  and  II  were 
tested  using  multiple  regression  to  determine  significant  factors  impacting  U  and  EOU. 

A  copy  of  the  questionnaire  used  in  this  study  is  found  in  the  appendix  to  this  thesis.  The 
questionnaire  appended  is  for  Word;  the  WordPerfect  questionnaire  was  identical  except 
that  it  referenced  WordPerfect  rather  than  Word. 
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rV.  Results 


Results 

The  results  of  the  procedures  described  in  the  preceding  chapter  are  reported  here. 
The  data  were  coded  and  reduced  for  analysis  as  appropriate  for  the  various  measurement 
instruments.  First,  data  for  the  two  groups  (Word  and  WordPerfect)  were  analyzed  and 
compared.  Finally,  data  for  the  two  groups  was  combined  into  a  single  data  set  which 
yielded  a  predictive  model  based  on  regression  analysis.  Data  reduction  and  analysis 
have  yielded  the  findings  encapsulated  in  Tables  1  through  8. 

Overall  Results.  The  results  shown  in  Tables  1  and  2  (below)  show  the  Pearson 
correlations  of  all  variables  and  all  dimensions  of  all  variables  for  the  two  groups  (Word 
for  Windows  and  WordPerfect,  respectively).  The  variables  QUIS  and  TLX  are  each 
comprised  of  several  dimensions.  Tables  1  and  2  present  results  in  which  each  dimension 
is  visible.  For  clarity,  lines  have  been  added  between  the  variables,  and  dimensions  of 
variables  have  been  indented. 

The  variables  shown  include  Usefulness  (U),  Ease  of  Use  (EOU),  Questionnaire 
on  User  Interface  Satisfaction  (QUIS),  NASA  Task  Load  Index  (TLX),  Accuracy 
(ACCUR),  and  Time.  QUIS  is  comprised  of  five  dimensions;  Overall  Reaction 
(REACT),  Screen,  Terminology  (TERM),  Learning  (LEARN),  and  Systems  Capabilities 
(SYS).  TLX  is  comprised  of  six  dimensions;  Mental  Demand  (MD),  Physical  Demand 
(PD),  Time  Demand  (TD),  Performance  (PERF),  Effort  (EFF),  and  Frustration  Level 
(FRUS).  It  should  be  noted  from  Tables  1  and  2  that  the  dimensions  of  TLX  (measure  of 
subjective  mental  workload)  and  the  dimensions  of  QUIS  (measure  of  satisfaction)  are 
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highly  correlated  with  the  overall  variables  of  TLX  and  QUIS,  respectively.  An 
exception  to  the  this  high  level  of  correlation  is  found  is  the  relatively  low  correlations 
between  TLX  and  the  dimension  of  PD  (physical  demand).  The  low  correlation  is  not 
xmexpected  in  light  of  the  relatively  low  physical  demands  placed  on  the  participants, 
e.g.,  using  a  desk  top  computer  located  in  a  computer  laboratory.  However,  the  overall 
picture  suggests  that  these  measures  are,  in  fact,  measuring  a  single  (multi-dimensional) 
construct. 

Table  2.  Correlations  (Pearson)  of  Variables  and  Dimensions,  Word 


U 

EOU 

REACT 

SCREEN  TERM 

LEARN  SYS 

EOU 

0.4104 

REACT 

0.2473 

0.3834 

SCREEN 

0.1033 

0.4416 

0.2296 

TERM 

0.0587 

0.4135 

0.3188 

0.8358 

LEARN 

0.1538 

0.5175 

0.4217 

0.6027 

0.6818 

SYS 

0.1993 

0.4314 

0.5336 

0.4825 

0.5662 

0.5118 

QUIS 

0.2091 

0.5580 

0.7236 

0.7683 

0.8334 

0.8028  0.7615 

MD 

-0.0930 

-0.4154  • 

■0.2636 

-0.0771 

-0.1838 

-0.1131  -0.1939 

PD 

-0.1242 

-0.2453 

0.1917 

-0.0889 

-0.1956 

-0.0824  -0.0444 

TD 

-0.2938 

-0.3745 

-0.1158 

-0.0379 

-0.0644 

-0.2043  -0.2185 

PERF 

-0.3174 

-0.5230 

-0.2195 

-0.1861 

-0.2598 

-0.3762  -0.2008 

EFF 

-0.1288 

-0.2927 

-0.3390 

-0.0900 

-0.2147 

-0.1971  -0.0531 

FRUS 

-0.3649 

-0.5203 

-0.1960 

-0.0824 

-0.0609 

-0.0352  -0.1367 

TLX 

-0.2723 

-0.4537 

-0.2252 

-0.0959 

-0.2047 

-0.1935  -0.1717 

ACCUR 

0.1791 

0.3178 

-0.2021 

-0.0174 

-0.0560 

0.0164  -0.1718 

TIME 

-0.2301 

-0.3713 

-0.2060 

-0.2849 

-0.2558 

-0.1202  -0.1593 

MD 

PD 

TD  PERF  EFF  FRUS 

MD 

-0.2283 

PD 

-0.0191 

0.2638 

TD 

-0.1559 

0.5531 

0.3621 

PERF 

-0.3182 

0.4154 

0.2680 

0.3984 

EFF 

-0.2642 

0.7104 

0.0809 

0.5399 

0.4246 

FRUS 

-0.1457 

0.5661 

0.2774 

0.5065 

0.3373 

0.4897 

TLX 

-0.2373 

0.8258 

0.2774 

0.7517 

0.6781 

0.8381  0.6527 

ACCUR 

-0.1245 

0.0244 

-0.0902 

-0.0254 

-0.2329 

-0.0549  -0.1674 
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TIME 

-0.2713 

0.2789  0.2376  0.3097  0.3336  0.3306  0.5015 

TLX 

ACCUR 

ACCUR 

-0.1461 

TIME 

0.3995 

0.0285 

Table  3. 

Correlations  (Pearson)  of  Variables  and  Dimensions,  WordPerfect 

■  U 

EOU 

REACT  SCREEN  TERM  LEARN  SYS 

EOU 

0.2305 

REACT 

0.5132 

0.5751 

SCREEN 

0.4681 

0.2129 

0.5002 

TERM 

0.5993 

0.5157 

0.6980 

0.6024 

LEARN 

0.3481 

0.6568 

0.5275 

0.3938  0.6130 

SYS 

0.2815 

0.3715 

0.5361 

0.5395  0.5072  0.6782 

QUIS 

0.5591 

0.5655 

0.8202 

0.7758  0.8557  0.7695 

0.7977 

MD 

0.0748 

-0.3456 

-0.1278 

0.0367  -0.2696  -0.0726 

0.1842 

PD 

-0.1035 

-0.1167 

-0.0245 

-0.3862  -0.0161  0.1226 

0.0431 

TD 

-0.0796 

-0.3663 

-0.2131 

-0.2062  -0.2533  -0.0280 

0.0929 

PER 

0.1030 

-0.3444 

-0.3263 

-0.1944  -0.1515  -0.1273 

-0.0950 

EFF 

0.0832 

-0.2904 

-0.0347 

0.0414  -0.0504  -0.0405 

0.1356 

FRUS 

-0.2113 

-0.5817 

-0.5658 

-0.3441  -0.4887  -0.2610 

-0.0430 

TLX 

0.0339 

-0.4769 

-0.2388 

-0.0761  -0.2242  -0.0656 

0.1009 

ACCUR 

0.2374 

0.4801 

0.4765 

0.1862  0.2391  0.2137 

0.2110 

TIME 

0.1811 

-0.2981 

-0.2876 

0.1380  -0.1086  -0.0047 

-0.0030 

QUIS 

MD 

PD 

TD  PER  EFF  FRUS 

MD 

-0.0641 

PD 

-0.0843 

0.1627 

TD 

-0.1637 

0.5340 

0.6597 

PER 

-0.2289 

0.4377 

0.0529 

0.2435 

EFF 

0.0134 

0.7896 

0.1462 

0.3996  0.6258 

FRUS 

-0.4361 

0.5624 

0.4337 

0.6312  0.4488  0.4505 

TLX 

-0.1322 

0.8078 

0.3354 

0.6522  0.7192  0.8727 

0.7509 

TEXT 

0.3342 

-0.1006 

-0.2736 

-0.4420  -0.2892  -0.1151 

-0.5302 

TIME 

-0.0680 

0.4211 

0.0593 

0.4979  0.3308  0.2259 

0.3566 
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Tables.  (Continued) 


_ TLX  TEXT 

TEXT  -0.4015 

TIME  0.4249  -0.4473 


Subsequent  presentation  of  correlations  provides  information  on  TLX  and  QUIS 
as  overall  variables,  without  specific  reporting  on  the  internal  dimensions.  Given  the 
high  correlations  among  the  internal  dimensions  of  the  QUIS  and  TLX,  the  analyses 
which  follow  use  only  the  overall  measures  of  satisfaction  (QUIS)  and  subjective  mental 
workload  (TLX),  respectively. 

Table  4.  Correlations  (Pearson)  of  Variables,  Word 


U  EOU  QUIS  TLX  ACCUR 
EOU  0.4104 

QUIS  0.2091  0.5580 

TLX  -0.2723  -0.4537  -0.2373 

ACCUR  0.1791  0.3178  -0.1245  -0.1461 

TIME  -0.2301  -0.3713  -0.2713  0.3995  0.0285 


The  correlations  shown  in  Table  3  provide  several  results  of  interest.  The 
correlation  between  QUIS  and  EOU  may  indicate  the  role  that  satisfaction  plays  in 
forming  EOU  perceptions.  As  expected,  mental  workload  (TLX)  correlated  negatively 
with  U  and  EOU,  suggesting  that  as  mental  workload  increased,  perceptions  of  usefulness 
and  ease  of  use  decreased.  Additionally,  a  negative  correlation  between  TIME  (time 
required  for  the  task)  EOU  was  observed.  This  negative  correlation  indicates  that  as  the 
time  required  to  complete  the  task  increased,  the  formulation  of  perceived  EOU 
decreased. 
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Table  5.  Correlations  (Pearson)  of  Variables,  WordPerfect 


U  EOU  OUIS  TLX  ACCUR 
EOU  0.2305 

QUIS  0.5591  0.5655 

TLX  0.0339  -0.4769  -0.1322 

ACCUR  0.2374  0.4801  0.3342  -0.4015 

TIME  0.1811  -0.2981  -0.0680  0.4249  -0.4473 

The  results  for  WordPerfect  (Table  4)  are  similar  to  those  for  Word.  One 
difference  in  Table  4  is  the  strong  relationship  between  QUIS  and  U.  This  may  indicate  a 
user  preference  for  the  interface  or  the  features  of  this  particular  word  processor.  The 
version  of  WordPerfect  used  in  this  study  was  text  based,  required  keystrokes  for 
functions,  and  did  not  implement  the  modem  standard  of  WYSIWYG  (what  you  see  is 
what  you  get)  so  that  the  document  as  it  would  be  printed  as  it  appeared  on  the  monitor. 

In  contrast.  Word  used  icon  menus,  graphical  interface,  and  featured  WYSIWYG.  Both 
word  processors  featured  standard  editing  functions,  and  special  tools  such  as  spell 
checking. 

Also,  a  significant  correlation  exists  between  accuracy  (ACCUR)  and  EOU.  This 
result  is  not  surprising,  but  did  not  occur  in  the  data  for  Word  (Table  3).  Finally,  the  data 
for  WordPerfect  (Table  4)  indicates  a  significant  negative  relationship  between  time  and 
accuracy.  In  other  words,  the  more  time  the  participant  required,  the  less  accurate  the 
work.  This  is  somewhat  surprising;  one  possible  explanation  would  be  that  some 
participants  had  difficulties  using  WordPerfect,  or  some  participants  spent  time  manually 
searching  for  and  correcting  errors. 
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Because  the  correlation  results  indicated  some  differences  between  the  data  for  the 
different  word  processors,  the  researcher  conducted  further  analysis  of  the  data  to 
determine  whether  there  were  mean  differences  between  the  two  word  processors  across 
each  of  the  variables.  Table  5  depicts  a  summary  of  the  t-test  results  of  the  six  principal 
variables. 

Table  6.  t-Test  Results 


Variable 

Word 

MeanrVariance) 

WordPerfect 

MeanrVariance) 

I 

E 

U 

5.85(0.95) 

5.22(0.75) 

-2.48 

0.02 

EOU 

5.81(1.01) 

4.64(2.01) 

-3.65 

<0.01 

QUIS 

6.82(1.26) 

5.40(1.42) 

-4.71 

<0.01 

TLX 

38.96(295.27) 

47.87(388.34) 

1.83 

0.08 

ACCUR 

7.12(2.19) 

5.52(4.01) 

-3.29 

<0.01 

TIME 

35.76(68.94) 

39.72(63.54) 

1.83 

0.08 

The  p  values  reported  in  table  5  are  for  the  two-tail  test.  The  p  values  for  the 
variables  U,  EOU,  QUIS  and  ACCUR  are  significant  at  the  0.05  level.  This  indicates  that 
there  were  no  significant  mean  differences  across  these  measures.  Both  TLX  and  TIME 
exhibit  non-significant  p  values  (at  the  0.05  level),  but  they  approach  significance. 

Given  the  differences  between  the  data  for  the  two  groups,  the  next  step  was  to 
identify  which  of  the  variables  offered  the  most  exploratory  power  in  predicting  EOU. 
This  was  accomplished  by  use  of  multiple  regressions  for  both  Word  and  WordPerfect. 
EOU  was  entered  as  the  dependent  variable  (DV),  and  each  of  the  variables  U,  QUIS, 
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TLX,  ACCUR,  and  TIME  were  entered  as  the  independent  variables  (TV).  The  method  of 
regression  was  unweighted,  least  squares,  linear  regression.  The  results  of  these 
regressions  are  presented  below.  Tables  6  and  7  show  the  regression  models  of  the  two 
samples.  Word  and  WordPerfect,  and  Table  8  shows  the  regression  model  for  the 
combined  samples. 


Table  7.  Linear  Regression  of  EOU,  Word 


PREDICTOR 

VARIABLES  COEFFICIENT  STD  ERROR  T _ P 


CONSTANT 

1.39060 

1.44107 

0.96 

0.3420 

U 

0.19437 

0.14492 

1.34 

0.1896 

QUIS 

0.41324 

0.10798 

3.83 

0.0006 

TLX 

-0.01027 

0.00706 

-1.45 

0.1561 

ACCUR 

0.22081 

0.08339 

2.65 

0.0126 

TIME 

-0.01525 

0.01491 

-1.02 

0.3143 

R-SQUARED 

0.5775 

RESID.  MEAN  SQUARE  (MSE) 

0.50745 

ADJUSTED  R-SQUARED 

0.5094  STANDARD  DEVIATION  0.71235 

SOURCE 

DF 

SS 

MS  F 

P 

REGRESSION 

5 

21.5023 

4.30045  8.47 

0.0000 

RESIDUAL 

31 

15.7308 

0.50745 

TOTAL 

36 

37.2331 

Table  8.  Linear  Regression  of  EOU,  WordPerfect 


PREDICTOR 


VARIABLES 

COEFFICIENT 

STD  ERROR 

T 

P 

CONSTANT 

2.65602 

1.94621 

1.36 

0.1883 

U 

-0.12951 

0.33424 

-0.39 

0.7027 

QUIS 

0.59593 

0.23711 

2.51 

0.0211 

TLX 

-0.02294 

0.01315 

-1.75 

0.0971 

ACCUR 

0.13440 

0.14160 

0.95 

0.3544 

TIME 

-0.00514 

0.03491 

-0.15 

0.8844 

R-SQUARED  0.5178  RESID.  MEAN  SQUARE  (MSE)  1.22161 
ADJUSTED  R-SQUARED  0.3909  STANDARD  DEVIATION  1.10527 
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Tables.  (Continued) 


SOURCE  DF 

SS 

MS 

F 

P 

REGRESSION 

5 

24.9244 

4.98487 

4.08  0.0110 

RESIDUAL 

19 

23.2106 

1.22161 

TOTAL 

24 

48.1350 

Table  9.  Linear  Regression  of  EOU,  Word  and  WordPerfect  (Combined) 


PREDICTOR 

VARIABLES  COEFFICIENT  STD  ERROR  T  P 


CONSTANT 

2.19768 

1.23601 

1.78 

0.0826 

U 

0.07794 

0.15151 

0.51 

0.6097 

QUIS 

0.52296 

0.10409 

5.02 

0.0000 

TLX 

-0.01342 

0.00755 

-1.78 

0.0828 

ACCUR 

0.14982 

0.07596  1.97 

0.0552 

TIME 

-0.02567 

0.01798 

.-1.43 

0.1608 

R-SQUARED  0.6432  RESID.  MEAN  SQUARE  (MSE)  0.74696 
ADJUSTED  R-SQUARED  0.6008  STANDARD  DEVIATION  0.86427 

SOURCE _ DF _ SS  MS _ F  P 

REGRESSION  5  56.5639  11.3128  15.15  0.0000 

RESIDUAL  42  31.3723  0.74696 

TOTAL  47  87.9362 


The  regression  model  for  the  combined  data  yields; 

EOU=0.078U  +  0.523QUIS  -  0.013TLX  +  0.150ACCUR  -  0.026TIME  +  2.198 
This  equation  shows  that  QUIS  is  the  strongest  individual  factor,  followed  by  ACCUR. 
This  fitiHing  is  supported  by  the  regressions  for  each  of  the  two  samples,  where  QUIS  is 
significant  and  ACCUR  was  significant  in  Word  sample.  The  overall  regression  indicates 
that  ACCUR  makes  a  meaningful  contribution  to  the  model,  and  with  p=0.055  is  on  the 
boundary  of  significance.  QUIS  is  clearly  significant  as  a  major  factor  in  the  model  and 
emerges  as  the  single  best  predictor  of  EOU.  Although  the  role  of  QUIS  outweighs  all 
others  in  the  model,  it  is  interesting  to  note  that  the  overall  model  delivers  R^  =  0.643.  In 
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other  words,  this  model  explains  64%  of  the  variance.  While  the  other  factors  in  the 
model  do  not  contribute  as  the  researcher  had  expected,  the  results  certainly  indicate  that 
satisfaction  (QUIS)  is  a  strong  predictor  of  EOU. 
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V.  Discussion 


General  Discussion 

The  research  this  thesis  reports  on  was  undertaken  to  improve  understanding  of 
user  acceptance  of  technology,  specifically,  information  technology.  A  significant  body 
of  research  in  HCI  has  accumulated  supporting  the  importance  of  effectiveness, 
efficiency,  and  satisfaction  as  determinants  of  usability.  Specifically,  researchers  in  HCI 
have  suggested  subjective  mental  workload,  accuracy,  and  time  for  task  completion  as 
potentially  useful  measures  for  predicting  system  usability.  Further,  MIS  researchers 
have  amassed  a  body  of  evidence  supporting  the  criticality  of  user  perceptions.  From  a 
qualitative  perspective,  the  notion  can  be  forwarded  that  HCI  has  accomplished  much  in 
response  to  the  question,  "Can  users  use  the  system?"  It  can  also  be  suggested  that  MIS 
has  responded  in  a  significant  manner  to  the  question,  "Will  users  use  the  system?"  The 
research  reported  has  drawn  fi'om  the  theoretical  framework  of  both  the  MIS  and  HCI 
fields,  and  attempted  to  find  the  theoretical  “common  ground”  that  lies  between  the  two 
disciplines.  Empirically,  instruments  used  extensively  in  HCI  research,  QUIS  and  TLX, 
were  applied  to  the  MIS  construct  of  perceived  EOU. 

The  research  question  that  motivated  this  researcher  focuses  on  the  formulation  of 
perceived  ease  of  use.  The  proposition  formulated  was: 

The  user’s  cognitive  experience  with  the  technology,  in  terms  of  effectiveness, 
efficiency,  and  satisfaction,  significantly  influences  the  formation  of  EOU 
perceptions. 
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The  results  only  partially  support  the  proposition.  The  data  do  not  support  the  anticipated 
role  of  efficiency  and  effectiveness  in  formulating  EOU.  Efficiency,  operationalized  as 
time  required  to  complete  task  and  subjective  mental  workload,  did  not  appear  to 
influence  EOU,  while  effectiveness,  operationalized  as  accuracy  in  completing  a  required 
task,  was  marginally  significant.  However,  satisfaction  emerged  as  a  very  significant 
determinant  of  EOU.  This  result,  though  somewhat  unexpected,  does  demonstrate  the 
existence  of  theoretical  constructs  that  may  be  shared  across  both  HCI  and  MIS. 

The  mixture  of  validated  instruments  accepted  in  both  the  HCI  and  MIS  arenas 
made  possible  the  construction  of  a  model  that  successfully  predicts  formulation  of  EOU. 
On  a  positive  note,  the  full  model  (including  effectiveness,  efficiency,  and  satisfaction) 
was  able  to  account  for  64%  of  the  variance  in  EOU.  While  individual  “pieces”  were  not 
always  significant,  the  entire  model  did  provide  a  good  fit  to  the  data.  As  mentioned 
earlier,  the  most  important  predictor  was  satisfaction,  measured  by  the  QUIS,  which 
accounted  for  52%  of  the  variance  in  EOU. 

The  data  indicate  that  efficiency  and  effectiveness  do  not  play  a  significant  role  in 
EOU  formulation.  One  explanation  for  this  result  could  be  that  these  results  were 
affected  by  factors  peculiar  to  this  study,  such  as  the  dissimilar  user  interfaces  used  by  the 
two  groups,  or  self-selection  variables  associated  with  the  sample  groups.  (Word 
employed  the  modem  WYSIWYG  standard  and  a  graphical  interface.  WordPerfect 
employed  a  text  based,  keystroke,  non- WYSIWYG  interface). 

The  fundamental  problem  approached  by  this  research  is  that  of  predicting  system 
usability  and  overall  acceptability  by  users.  To  date,  no  single  factor  has  emerged  as  a 
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reliable  predictor  of  acceptance.  Yet  the  results  reported  here  suggest  that  satisfaction 
should  be  considered  as  a  significant  factor  in  future  research. 

Implications 

The  significance  of  satisfaction  in  forming  EOU  has  theoretical  implications. 
Interestingly,  satisfaction  has  not  previously  been  incorporated  into  the  most  vvidely 
known  model  of  technology  acceptance  in  the  MIS  literature,  TAM  (see  Davis  and 
others,  1989).  This  research  suggests  that  extending  TAM  to  include  satisfaction  as  an 
important  construct  should  be  considered.  The  acceptance  of  satisfaction  into  the 
technology  acceptance  nomological  net  may  augment  our  current  understanding  of  user 
acceptance. 

The  significance  of  these  findings  have  practical  implications  as  well.  MIS 
practitioners  often  struggle  to  find  and  use  instruments  that  will  accurately  predict 
acceptance  of  information  technology.  If  MIS  managers  are  able  to  predict  user 
acceptance  before  a  system  is  deployed,  they  can  take  additional  actions,  as  needed,  to 
enhance  user  acceptance.  These  actions  could  include  additional  training,  improved 
documentation,  or  other  actions  that  are  warranted.  The  evidence  in  this  study  suggests 
both  that  satisfaction  is  a  variable  that  may  be  useful,  and  that  the  HCI  instrument,  QUIS, 
may  be  useful  in  measuring  satisfaction.  This  has  significant  implications  for  the 
practitioner  making  usability  evaluations  with  limited  resources.  The  QUIS  is  relatively 
inexpensive  to  administer,  and  appears  to  reliably  predict  EOU. 

These  findings  may  also  be  useful  in  development  of  systems,  especially  within 
the  rapid-prototyping  approach,  in  which  prototypes  are  quickly  produced,  evaluated,  and 
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modified.  QUIS  could  provide  a  developer  with  a  quick-look  of  usability  prior  to  full 
commitment  to  a  particular  design. 

Limitations 

As  with  all  research,  there  are  a  number  of  limitations  inherent  in  this  study. 

First,  the  samples  used  for  the  study  were  drawn  from  undergraduates  at  a  large 
ximversity.  These  students  were  all  experienced  with  personal  computers,  in  general,  and 
word  processors,  specifically.  Different  results  may  be  obtained  with  samples  that  have 
less  experience  or  computer  knowledge.  Furthermore,  the  use  of  word  processors  as  a 
medium  may  not  generalize  to  other  software  applications.  Also,  the  task  performed  in 
the  study  was  a  relatively  short,  uncomplicated  task.  Advanced  techniques,  such  as 
graphics  integration,  tables,  etc.,  were  not  part  of  the  task.  A  more  complex  task  might 
cause  effectiveness  and  efficiency  to  be  more  salient  to  users  (and  thereby  significant  in 
prediction  of  EOU).  Finally,  because  of  the  self-selection  element  of  the  study,  the 
sample  size  for  the  WordPerfect  sample  was  25.  While  this  was  thought  to  be  adequate 
to  test  the  proposition  central  to  this  study,  a  larger  sample  may  have  yielded  greater 
statistical  power,  causing  those  variables  which  were  only  marginally  significant  to  reach 
statistical  significance  at  the  alpha  =  .05  level. 

Conclusion 

The  findings  of  this  study  are  significant  to  theorists  and  practitioners  alike.  The 
emergence  of  satisfaction,  as  a  powerful  predictor  of  EOU  is  notable.  Use  of  well 
accepted  HCI  and  MIS  instruments  together  to  build  a  data  model  that  predicts  EOU, 
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accounting  for  64%  of  the  variance  strongly  suggests  that  both  MIS  and  HCI  bring 
important  perspectives  that  are  useful  in  understanding  human  interaction  with 
information  technology.  The  use  of  HCI  instruments  to  obtain  data  useful  to  predicting 
EOU  may  be  a  first  step  in  building  a  bridge  between  the  HCI  perspective  of  "can  they 
use?"  and  the  MIS  perspective  of  "will  they  use?"  More  research  is  recommended  to 
explore  the  overlap  in  the  theoretical  landscape  that  lies  between  HCI  and  MIS. 
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Appendix:  Word  Processing  Software  Questionnaire 


Student  ID  Number 


Part  1:  For  questions  1-3,  please  circle  the  best  response. 

1.  Sex 

a)  Male 

b)  Female 

2.  How  long  have  you  been  using  Word  for  Windows? 

a)  Less  than  6  months 

b)  At  least  6  months,  but  less  than  1  year 

c)  At  least  1  year,  but  less  than  2  years 

d)  2  years  or  more 

3 .  On  average,  how  many  hours  per  week  do  you  use  Word  for  Windows? 

a)  Less  than  1  hoxir  per  week 

b)  At  least  1  hour,  but  less  than  5  hours  per  week 

c)  At  least  5  hours,  but  less  than  1 0  hours  per  week 

d)  10  hours  per  week  or  more 


Part  2:  For  questions  4-11  below,  please  circle  the  number  which  best  indicates 
your  opinion  about  the  software  that  you  just  used. 

4.  Using  Word  for  Windows  improves  my  performance  in  my  classwork. 

Stongly  +3  +2  +10  -1  -2  -3  Strongly 

Agree  Disagree 

5.  Using  Word  for  Windows  in  my  classwork  increases  my  productivity. 

Stongly  +3  +2  +1  0  -1  -2  -3  Strongly 

Agree  Disagree 
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Using  Word  for  Windows  enhances  my  effectiveness  in  my  classwork. 


Stongly  +3  +2  +1  0  -1  -2 

-3 

Strongly 

Agree 

Disagree 

I  find  Word  for  Windows  useful  in  my  classwork. 

Stongly  +3  +2  +1  0  -1  -2 

-3 

Strongly 

Agree 

Disagree 

I  find  it  easy  to  get  Word  for  Windows  to  do  what  I  want  it  to  do. 

Stongly  +3  +2  +1  0  -1  -2 

-3 

Strongly 

Agree 

Disagree 

My  interaction  with  Word  for  Windows  is  clear  and  understandable. 

Stongly  +3  +2  +1  0  -1  -2 

-3 

Strongly 

Agree 

Disagree 

Interacting  with  Word  for  Windows  does  not  require  a  lot  of  my  mental  effort. 

Stongly  +3  +2  +1  0  -1  -2 

-3 

Strongly 

Agree 

Disagree 

I  find  Word  for  Windows  easy  to  use. 

Stongly  +3  +2  +1  0  -1  -2 

-3 

Strongly 

Agree 


Disagree 


For  questions  Parts  3-7,  please  circle  the  number  that  best  corresponds  to  your 
opinion  about  the  software  you  just  used. 

PART  3:  Overall  User  Reactions 


Please  circle  the  numbers  which  most  appropriately  reflect  yom  impressions  about  using 
this  word  processing  system.  Not  Applicable  =  NA. 


3 . 1  Overall  reactions  to  the  system: 

3.2 

3.3 

3.4 

3.5 

3.6 


terrible  wonderful 

123456789  NA 

frustrating  satisfying 

123456789  NA 

dull  stimulating 

123456789  NA 

difficult  easy 

123456789  NA 

inadequate  power  adequate  power 

123456789  NA 

rigid  flexible 

123456789  NA 
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PART  4:  Screen 


4. 1  Characters  on  the  computer  screen 

4.2  Was  the  highlighting  on 
the  screen  helpful? 

4.3  Were  the  screen  layouts  helpful? 

4.4  Sequence  of  screens 


hard  to  read  easy  to  read 

123456789  NA 

not  at  all  very  much 

123456789  NA 

never  always 

123456789  NA 

confusing  clear 

123456789  NA 


PART  5:  Terminology  and  System  Information 

5  1  Use  of  terms  throughout  system  inconsistent  consistent 

123456789 

5.2  Does  the  terminology  relate  well  to  the 

work  you  are  doing?  unrelated  well  related 

123456789 

5.3  Messages  which  appear  onscreen  inconsistent  consistent 

123456789 

5.4  Messages  which  appear  on  screen  confusing  clear 

123456789 

5.5  Does  the  computer  keep  you  informed 

about  what  it  is  doing?  never  always 


123456789 


5.6  Error  messages  unhelpful  helpful 

123456789 


NA 

NA 

NA 

NA 

NA 

NA 
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PART  6:  Learning 


6.1  Learning  to  operate  the  system 

6.2  Exploration  of  features 
by  trial  and  error 

6.3  Remembering  names  and 
use  of  commands 


6.4  Can  tasks  be  performed  in  a 
straight-forward  manner? 

6.5  Help  messages  on  the  screen 

6.6  Supplemental  reference  materials 


difficult  easy 

123456789 


discouraging  encouraging 

123456789 


difficult  easy 

123456789 


never  always 

123456789 

confusing  clear 

123456789 

confusing  clear 

123456789 


PART  7:  System  Capabilities 

7.1  System  speed 

7.2  How  reliable  is  the  system? 

7.3  System  tends  to  be 

7.4  Correcting  your  mistakes 

7.5  Are  the  needs  of  both  experienced  and 
inexperienced  users  taken  into 
consideration? 


too  slow  fast  enough 

123456789 

imreliable  reliable 

123456789 

noisy  quiet 

123456789 

difficult  easy 

123456789 


never  always 

123456789 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 
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In  addition  to  the  questions  above,  we  would  also  like  to  know  about  the  workload  you 
experienced  in  performing  this  editing  task.  However,  feelings  of  workload  can  come 
from  several  different  factors.  For  example,  some  people  feel  that  mental  or  time 
demands  are  the  most  important  factors  in  perceived  workload.  Others  may  feel  that  their 
performance  or  amount  of  frustration  is  the  most  important  part  of  their  feelings  of 
workload. 

The  following  section  will  present  you  with  a  series  of  pairs  of  rating  scale  titles  (for 
example.  Effort  vs.  Mental  Demands).  You  will  be  asked  to  choose  which  of  the  items 
was  more  important  to  vour  experience  of  workload  in  the  editing  task  that  you  just 
performed.  Titles  and  descriptions  of  the  scale  titles  are  presented  below: 


Title 

Endooints 

Descriptions 

Mental  Demand 

Low/High 

How  much  mental  and  perceptual  activity 
was  required  (e.g.  thinking,  deciding, 
calculating,  remembering,  looking, 
searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or 
forgiving? 

Physical  Demand 

Low/High 

How  much  physical  activity  was  required 
(e.g.  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or 
demanding,  slow  or  brisk,  slack  or 
strenuous,  restful  or  laborious? 

Time  Demand 

Low/High 

How  much  time  pressure  did  you  feel  due  to 
the  rate  or  pace  at  which  the  tasks  occurred? 
Was  the  pace  slow  and  leisurely  or  rapid  and 
frantic? 

Performance 

Good/Poor 

How  successful  do  you  think  you  were  in 
accomplishing  the  oals  of  the  task?  How 
satisfied  were  you  with  your  performance  in 
accomplishing  these  goals? 

Effort 

Low/High 

How  hard  did  you  have  to  work  (mentally 
and  physically)  to  accomplish  your  level  of 
performance. 

Frustration  Level 

Low/High 

How  insecure,  discouraged,  irritated, 
stressed,  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent 
did  you  feel  during  the  task. 

You  will  be  using  the  scale  titles  illustrated  above  in  Parts  8  and  9  on  the  following 
pages.  You  may  refer  back  to  this  table  as  needed. 
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Part  8:  For  each  pair,  circle  the  scale  title  that  represents  the  more  important 
contributor  to  workload  for  the  editing  tasks  you  performed  in  this  session. 


8.01.  Effort  or  Performance 


8.02.  Time  Demand  or  Effort 


8.03.  Performance  or  Frustration 


8.04.  Physical  Demand  or  Performance 


8.05.  Time  Demand  or  Frustration 


Continued  on  the  following  page 


Continued  jfrom  previous  page:  for  each  pair,  circle  the  scale  title  that  represents  the 
more  important  contributor  to  workload  for  the  editing  tasks  you  performed  in  this 
session. 


8.06.  Physical  Demand  or  Frustration 


8.07.  Physical  Demand  or  Time  Demand 


8.08.  Time  Demand  or  Mental  Demand 


8.09.  Frustration  or  Effort 


8.10.  Performance  or  Time  Demand 


Continued  on  the  following  page 
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Continued  from  previous  page:  for  each  pair,  circle  the  scale  title  that  represents  the 
more  important  contributor  to  workload  for  the  editing  tasks  you  performed  in  this 
session. 


8.11.  Mental  Demand  or  Physical  Demand 


8.12.  Frustration  or  Mental  Demand 


8.13.  Performance  or  Mental  Demand 


8. 14.  Mental  Demand  or  Effort 


8.15.  Effort  or  Physical  Demand 
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Part  9;  For  questions  9.1  -  9.6,  place  an  “X”  on  each  scale  at  the  point  which 
matches  your  experience.  Each  line  has  two  endpoint  descriptors  that  describe  the 
scale.  Please  note  that  Performance  goes  from  “good”  on  the  left  side  of  the  scale  to 
“poor”  on  the  right.  Consider  each  scale  individually.  Your  ratings  will  play  an 
important  role  in  the  evaluation  being  conducted,  therefore,  your  participation  is 
greatly  appreciated. 

9.1.  Mental  Demand 


1 

1 

1 

1 

1 

_L 

_L 

_L 

1 

_L 

5h 

L 

1 

nij 

9.2.  Physical  Demand 


1 

1 

1 

1 

L 

1 

1 

_L 

1 

_L 

-I- 

L 

ow 

nign 

9.3.  Time  Demand 


9.4.  Performance 


9.5.  Effort 
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Frustration 
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Your  response  is  important  Thank  you. 

1.  Did  this  research  contribute  to  a  current  research  project?  a.  Yes  b.  No 


2.  Do  you  believe  this  research  topic  is  significant  enough  that  it  would  have  been  researched  (or 
contracted)  by  your  organization  or  another  agency  if  AFIT  had  not  researched  it? 

a.  Yes  b.  No 


3.  Please  estimate  what  this  research  would  have  cost  in  terms  of  manpower  and  dollars  if  it  had 
been  accomplished  under  contract  or  if  it  had  been  done  in-house. 

Man  Years _  $ _ 


4.  Whether  or  not  you  were  able  to  establish  an  equivalent  value  for  this  research  (in  Question 
3),  what  is  your  estimate  of  its  significance? 

a.  Highly  b.  Significant  c.  Slightly  d.  Of  No 

Significant  Significant  Significance 

5.  Comments  (Please  feel  fi-ee  to  use  a  separate  sheet  for  more  detailed  answers  and  include  it 
with  this  form): 


Name  and  Grade 


Organization 


Position  or  Title 


Address 


